Method and arrangement for entering data in an electronic apparatus and an 
electronic apparatus 

The invention relates to a method for entering, by means of a virtual keyboard, data 
in an electronic apparatus housed within a body, which method comprises the steps 
5 of emitting infrared radiation by at least one infrared transmitter in at least one 
plane, placing an obstacle in the radiation field of the infrared transmitter, receiving 
infrared radiation reflected from the obstacle by at least one infrared receiver in at 
least one plane, preprocessing the received reflected signal, and determining the 
position of the obstacle on the basis of the received preprocessed signal. The inven- 
10 tion fiirther relates to an input arrangement of an electronic apparatus for realizing 
the operation of the above-mentioned virtual keyboard, and to a recognition 
program belonging to the input arrangement. 

Various wireless devices, such as cellular phones and palmtop computers, need 
some kind of a physical user interface for data and command input. For example, it 

15 is known a small mobile phone keyboard with keys 1-9, *, 0 and #, in addition to 
other possible letter characters produced by these keys. Likewise, it is known a so- 
called "communicator" keyboard of a cellular phone used in cellular networks with 
alphanumeric keys a-z and 1-0. These keyboards are made as small as possible so 
that they are relatively difficult to use in comparison with keyboards of normal 

20 computers. ^ 

Touch keyboards of various sizes for electric devices are also known. Such key- 
boards are usually arranged in the form of a touchscreen on the device's display. In 
such an arrangement, arrays of LED transmitters and receivers, for example, are 
placed in the frame around the screen, on more than one side of the screen. When a 
25 finger or some other obstacle is pointed at a position on the screen, a ray of infrared 
light is cut off, resulting in some predetermined action. 

Patent application FI 990676 fiirther discloses a method in which there is generated, 
by means of a plurality of infrared transceiver units, an artificial, virtual, two- 
dimensional keyboard arrangement on one side of an apparatus. In the close vicinity 
30 of the apparatus there is thus created a data input area, a kind of a two-dimensional 
virtual keyboard in the X-Y plane. According to the said application, an image of 
the virtual keyboard can be advantageously projected on an even surface by means 
of a laser diode and refractive optics. The said surface on which the virtual key- 
board is generated is usually the surface of a table. The exact position of an 
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obstacle/pointer or finger on the plane determined by the virtual keyboard is 
detected by means of infrared transmitters and receivers. The exact position on the 
virtual keyboard detected by the device disclosed corresponds to the execution of a 
certain function or to the pressing of a key on a real keyboard. 

5 Fig. 1 shows in top and lateral views the fundamental construction of the device 
mentioned above. The virtual keyboard arrangement comprises an apparatus 10 
which includes a display 11 and a virtual keyboard 13. The apparatus 10 is placed 
on the surface of a table 16, for example. The apparatus and its virtual keyboard 13 
are activated. The virtual keyboard 13 comprising a text grid is projected from the 

10 apparatus 10 onto the surface 16 by means of light sources and suitable optics. It is 
also possible to use a planar plate or board on the surface of which the key positions 
are marked. The apparatus 10 disclosed in the above-mentioned document com- 
prises in the example depicted in Fig. 1 six infrared transceivers 12a to 12f In this 
arrangement each vertical column of the keyboard has a transceiver unit of its own. 

15 For each column a narrow infrared light beam 14a is emitted. The light beam 14a 
emitted is reflected from an obstacle 15, say a finger or a pointer like a pen, back 
into the apparatus 10. The reflected light beam 14b is received by one or more 
receivers 12a to 12f inthe apparatus 10. 

In the example of Fig. 1 the virtual keyboard is drawn to consist of bright horizontal 

20 and vertical lines as well as letters, numbers and possible special characters such as 
and The lines represent the borders between the individual keys. The 
optical technology for projecting the virtual keyboard is well known. Using digital 
signal processing (DSP) techniques it can be calculated from the intensity or angle 
of incidence of the scattered infrared light or from the phase angle of the signal, the 

25 position of the obstacle 15 on the surface 16 in the same X-Y plane as the bottom 
plane of the apparatus. The problem with this arrangement is, however, that the 
received signal has a very nonlinear nature because it is reflected from an irregular- 
shaped obstacle. In addition, randomly varying infrared radiation is received from 
the surroundings, adding to the noisiness of the received signal. Therefore, the 

30 amplitude and phase characteristics of the signal reflected/returned back to the 
receiver vary a great deal not just on the basis of the position of the obstacle. Thus, 
various tabulated correction coefficients, which have been obtained by experimen- 
tation, have to be used to calculate the position of the obstacle. Therefore, the 
computational processing of the received signal is difficult and requires large com- 

35 puting capacity in order to calculate the position of the obstacle at the desired 
accuracy from the received signal. In this case problems will be caused by the 
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computing capacity of the processor and the size of the memory needed in the cal- 
culation in a cellular phone, for example. Moreover, if the position of the obstacle is 
to be determined in a third dimension as well (X, Y, and Z dimensions), the calcu- 
lation will require very much computing capacity and memory, and to realize such a 
5 solution by conventional DSP techniques in a small handheld device is difficult. 

An object of this invention is to provide a new kind of decision-making procedure 
and arrangement for a virtual keyboard arrangement, which requires less computing 
capacity and memory than an arrangement implemented using DSP solutions 
according to the prior art. Another object of this invention is to make possible the 
10 creation of a three-dimensional artificial keyboard/user interface. The method and 
arrangement can be advantageously applied in fiiture third-generation cellulsir 
terminals. 

The objects of the invention are achieved by an arrangement realizing a virtual two- 
or three-dimensional keyboard arrangement/user interface functioning in the infi-a- 
15 red region. The nonlinear and multidimensional signal obtained from this virtual 
keyboard is processed using an arrangement based on a pattern recognition method, 
such as e.g. a neural network technique efficient at processing nonlinear signals. 

The method for recognizing the position of an obstacle in a virtual keyboard is 
characterized in that to determine the position of the obstacle in at least one plane, a 
20 neural network arrangement is used into which a preprocessed signal is conducted 
to determine the position. 

The input arrangement of a virtual keyboard according to the invention is charac- 
terized in that the input arrangement comprises a pattern-recognizing neural 
network element stored in the memory of the apparatus to recognize the use of a 
25 given key in the virtual keyboard. 

The program realizing the recognition of the operation of the virtual keyboard is 
characterized in that the program comprises a step of receiving input signals in at 
least one neuron of a hidden layer of a neural network, a step of weighting input 
signals by suitable weighting coefficients in each hidden-layer neuron, a step of 
30 adding up the weighted input signals in each hidden-layer neuron in order to 
produce an output signal, a step of processing the output signal in the activation 
element of each hidden-layer neuron in order to produce a result signal, a step of 
processing the result signals of the hidden-layer neurons in the output cells of the 
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neural network, and a step of implementing the results of the inference routines of 
the neural network element. 

Some advantageous embodiments of the invention are specified in the dependent 
claims. 

5 The basic idea of the invention is as follows: A virtual keyboard is generated by 
means of an infrared transmitter-receiver arrangement beside an electronic appara- 
tus which advantageously is a cellular terminal. An obstacle, i.e. pointer, finger or 
the like, is placed in the infrared field thus generated, reflecting back infrared light 
emitted. The reflected infrared light is received by one or more infrared receivers. 

10 The peak value of every received signal is then determined, whereafter the signals 
are AD-converted, after which each received signal is low-pass filtered by a digital 
filter associated with it, and finally the filtered signals are conducted to a pattem 
recognition process which is advantageously a neural network arrangement. On the 
basis of the input signals (the filtered infrared receiver signals) the pattem 

15 recognition arrangement decides which key on the virtual keyboard the obstacle 
points at. 

An advantage of the invention is that in spite of the nonlinearities in the received 
signals the neural network arrangement used in the pattem recognition makes with 
high probability a correct decision compared with decisions made using DSP 
20 methods according to the prior art. 

Another advantage of the invention is that the terminal's computing power and size 
of memory required by the neural network arrangement used in the pattem recogni- 
tion are small compared with DSP methods according to the prior art. 

A fiirther advantage of the invention is that the method is not limited to two 
25 dimensions only but it can be supplemented with a positioning fiinction in a third 
dimension. 

The invention is described in detail in the following. The description refers to the 
accompanying drawings in which 

Fig. 1 shows top and side views of a virtual keyboard according to the prior art, 
30 Fig. 2a shows by way of example a cell in a neural network, 

Fig. 2b shows by way of example a neural network comprising a plurality of 
cells, 
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Fig. 3 shows by way of example the main components of an apparatus accord- 
ing to the invention, 

Fig. 4 shows by way of example the main components of an infrared 
transmitter-receiver arrangement, 
5 Fig. 5 shows by way of example the main components in signal preprocessing, 
Fig. 6 shows by way of example the main components of a neural network in 

the arrangement according to the invention, 
Fig. 7 shows by way of example a three-dimensional virtual keyboard accord- 
ing to the invention, and 
10 Fig. 8 shows a cellular terminal according to the invention and its connection 
with a cellular radio system. 

Fig. 1 was discussed in conjunction with the description of the prior art. 

Fig. 2a shows by way of example the principal components of a neural network cell 
15 20, hereinafter called a neuron, in a neural network arrangement used in pattern 
recognition. Data is input to an individual neuron via input lines. In the exemplary 
situation depicted in Fig. 2a, the values of five input signals (XO, XI, X2, X3, X4) 

21 are input to the neuron. Each of the individual values of the input signal 21 is 
weighted by a weighting coefficient (WO, Wl, W2, W3, W4) 22 corresponding to 

20 the input signal, and the input signal values thus weighted are added up in a 
summing element 23 associated with the neuron. The output signal Y, reference 
numeral 24, from the summing element 23 is conducted to the input of an activation 
element 25 in the neuron. The connection between the input and output of the 
activation element is determined by a fiinction f(Y). This function f(Y) is deter- 

25 mined/selected case by case and it may be either linear or nonlinear. The value of 
the result signal Z, reference numeral 29, at the output of the activation element 
depends of course on the signals 21 input to the neuron, the weighting coefficients 

22 applied, and on the activation ftinction 25 used. Training a neuron for a given 
operating model means that the weighting coefficients 22 and activation function 25 

30 are defined in the neuron in such a manner that a known input signal 21 produces a 
desired result signal 29. 

Fig. 2b shows by way of example one possible way of building a neural network. 
The neural network 200 depicted in Fig. 2b is a so-called multiplayer-perception 
(MLP) network. When training such a neural network by using e.g. back- 
35 propagation, one must know both the data coming to the neural network and the 
desired outcome of the inference of the neural network. On the basis of these, the 
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MLP network can be trained to realize the desired operating model/inference 
routine. 

The exemplary neural network in Fig. 2b comprises the network input signals XO, 
XI and X3, reference numeral 21a, the so-called hidden layer 27, and the neural 
5 network output cells 28. In the example depicted in Fig. 2b the hidden layer 27 and 
the output cells 28 both comprise three neural network neurons 20a, 20b. For 
simplicity. Fig. 2b does not show the weighting coefficients used in the weighting 
of the connections/signals 21a, 26a between the input signals and the different 
neurons in the neural network. The goal in the example depicted in Fig. 2b may be 

10 to train the neural network to produce result signal ZO if the input signal is XO, 
result signal Zl if the input signal is XI, and result signal Z2 if the input signal is 
X2. Naturally, all other signal combinations alone and in combination with each 
other are possible, too. In addition to the hidden layer shown in Fig. 2b the neural 
network may have other hidden layers connected one after the other or in parallel. 

1 5 The actual practical construction of the neural network depends on the intended use 
thereof. 

Fig. 3 shows by way of example the main functional components of a virtual key- 
board based on neural network technology according to the invention. An infrared 
transceiver 31, hereinafter IR transceiver, emits and receives infrared radiation 30. 

20 The received signal is preprocessed in block 32. In the example of Fig. 3, there are 
as many IR transceivers 31 and signal preprocessing blocks 32 as there are trans- 
ceivers 12a-12f in the example of Fig. 1. Naturally, the quantity of transceivers 12 is 
not limited to the six shown in Fig. 1. From block 32 the preprocessed signals are 
fed into the neural network element 33 which outputs signal 34 which advanta- 

25 geously corresponds to the pressing of a key on the virtual keyboard or a desired 
function of the terminal. 

Fig. 4 shows by way of example the main components of the IR transceiver 31a 
shown in Fig. 3. The transceiver comprises an IR sensor part 41 which in tum com- 
prises an IR trcmsmitter 42 to emit an IR beam 30a. It further comprises an IR 

30 receiver 41 to receive an incoming IR beam. The receiver 41 further comprises a 
received signal peak value hold circuit 44 which for a desired time keeps in its 
memory the received signal peak value which varies advantageously from 0 to 
3.5 volts. The analog signal peak value 46 is further conducted to signal pre- 
processing. Timing for the transmitter and receiver is provided by an oscillator 45 

35 which gives the clock signal 47 needed by the arrangement and which is also used 
in other timing of the operation of the input arrangement. 
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Fig. 5 shows by way of example the main components of the signal preprocessing 
part 32a in the arrangement according to Fig. 3. The received signal peak value 46 
and clock signal 47 are provided by the IR transceiver 31a. This peak value 46 
includes various unwanted components in addition to the actual payload signal. One 
5 of these is background radiation for which the preprocessing part 32a comprises a 
background radiation compensation circuit 54. The clock signal 47 is conducted to a 
trigger circuit 50. The trigger circuit controls signal sampling and the operation of 
an AD converter circuit 5 1 following the sampling. The AD converter produces a 
train of digital samples of the received signal, which is conducted to a digital low- 

10 pass filter 52. The upper frequency limit of the low-pass filter is advantageously 5 
to 20 Hz. The digital filter may be realized using either IIR (Indefinite Impulse 
Response) or FIR (Finite Impulse Response) technology. The low-pass filtered 
signal 53 is conducted to the neural network for the inference process. There are 
advantageously as many signals 53 as there are actual IR transceiver units in the 

1 5 apparatus according to the invention. 

Fig. 6 shows by way of example an advantageous embodiment to realize the neural 
network element 33 shown in Fig. 3. The neural network element 33 comprises a 
signal input part 61 into which all input signals 53 processed by the signal pre- 
processing part 32 are fed. There are advantageously as many of these input signals 

20 as there are IR transceivers in the apparatus according to the invention. The neural 
network element 33 proper comprises N discrete neural networks 62. N is advanta- 
geously greater than the number of IR transceivers in the apparatus. Advanta- 
geously N is at least equal to the number of separate keys/functions in the desired 
virtual keyboard. From the input signals 30b the neural network element 33 infers, 

25 in accordance with the inference rules it has learned, that a certain key Zm of the 
virtual keyboard was pressed. In this example, m is in the range 1 to N. An infer- 
ence system based on neural network technology according to this example requires 
10 to 50kB of memory, so it can be implemented in a cellular phone's memory. 

Fig. 7 shows by way of example an embodiment according to the invention which 
30 enables 3D positioning of an object in addition to the 2D positioning described 
above. The exemplary situation of Fig. 7 involves a so-called laptop computer 70. 
The latter comprises the body 71 of the apparatus, on a surface of which there is the 
keyboard part 74 proper. The cover part 72 of the apparatus includes the display 76 
proper. In accordance with the invention, IR transceivers 73 a, 73b are attached to 
35 both the body part 71 and the cover part 72. Now, in the exemplary situation of Fig. 
7, these IR transceivers 73a, 73b form a 3-dimensional network above the keyboard 
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74 proper. Signals from this 3 -dimensional network are processed by two neural 
networks according to the invention for the processing of a 2-dimensional signal. 
When an artificial obstacle, say a finger, pen or pointer, is placed above the key- 
board 74, the IR receivers of both the body part 71 and the cover part 72 recognize 
5 its position. When the angle O 75 between the body part 71 and cover part 72 is 
known, the position/shape of the obstacle can be computed 3-dimensionally from 
these two 2-dimensional measurements. An arrangement according to this embodi- 
ment may be advantageously utilized in the 3D manipulation of an object on the 
display 76 proper. Thus the 3 -dimensional pattem recognition arrangement accord- 
10 ing to the invention may be used instead of various commands issued by means of a 
keyboard or PC mouse. 

The virtual keyboard arrangement according to the invention may also be generated 
using a separate auxiliary, both in 2-dimensional and 3-dimensional embodiments. 
Such an auxiliary is connected with the terminal proper via either a wired or wire- 
15 less link. 

Above it was mainly described how the virtual keyboard according to the invention 
fiinctions as a substitute for a physical keyboard. However, a pattem recognition 
system according to the invention which is realized by a neural network has enough 
capacity to determine the position of a moving artificial obstacle, too. Therefore, the 

20 method according to the invention can be advantageously applied instead of a so- 
called mouse or various scrolling keys for controlling a display or issuing com- 
mands in 2D systems. When the invention is applied in a 3D device, the movement 
or stopping of the movement of an artificial obstacle may be used to realize versa- 
tile fiinctions. In particular, in 3D imaging the invention may be advantageously 

25 used to rotate an object on the screen, change its shape or color it, for example. 
Furthermore, it is possible to use the artificial obstacle as a three-dimensional means 
of drawing a 3-dimensional object. The arrangement according to the invention can 
also simultaneously recognize several artificial obstacles in the virtual keyboard and 
follow their movements. 

30 Fig. 8 shows a simplified block diagram of a cellular terminal 800 in which the vir- 
tual keyboard according to the invention can be applied. The terminal comprises an 
anterma 801 for receiving radio-frequency (RF) signals transmitted by base stations. 
A received RF signal is directed by a switch 802 to a RF receiver 806 where the 
signal is amplified and converted digital. The signal is then detected and demodu- 

35 lated in block 807. Block 808 performs decryption and deinterleaving. Then follows 
signal processing in block 809. Received data may be stored as such in the mobile 
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station's memory 804 or, alternatively, the processed packet data are taken after the 
signal processing to a possibly external device such as a computer. A control unit 
803 controls the above-mentioned reception blocks in accordance with a program 
stored in the unit. 

5 Transmission from the mobile station is performed e.g. as follows. Controlled by 
the control block 803, block 811 performs possible signal processing on the data 
and block 812 performs interleaving and encryption on the processed signal to be 
transmitted. Bursts are generated from the encoded data, block 813, which are 
modulated and amplified into a RF signal to be transmitted, block 814. The RF 
10 signal to be transmitted is conducted to the antenna 801 by means of the switch 802. 
These processing and transmission fiinctions, too, are controlled by the control unit 



In the terminal of Fig. 8, the components that are essential from the invention's per- 
spective include the prior-art control block 803 and the terminal's memory 804. In 

15 addition, the terminal should have an IR transceiver unit 805 according to the 
invention for producing a virtual keyboard 815. The control unit 803 controls the 
operation and use of the virtual keyboard according to the invention. Part of the 
memory 804 of the terminal has to be allocated for the generation of a neural net- 
work required by the operation of the virtual keyboard according to the invention. 

20 Naturally, a detachable memory medium associated with the terminal, such as a 
SIM card, which is known as such, may also be used for this same purpose. 

Above it was described some advantageous embodiments according to the inven- 
tion. The invention is not limited to the solutions described here. The inventional 
idea may be applied in many ways within the scope defined by the claims attached 
25 hereto. 
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